The only reason Elders get away with this kind of crap is because enough people allow it it ,and don't stand up to them .
You first need to have an understanding with your wife as others have stated above . She should not share family business with these men about you . It is in their own publications that the husband is the head of the household ,even IF he is an unbeliever .They may use the excuse they are looking out for her spiritual interests ,but this is a lie because they could do that without interrogating her about you . You need to immediately contact them and forbid this kind of behavior . Call HQ if necessary stating these men are causing division within the household and are over-stepping the bounds with your wife .
Even if they argue that they are only sheparding her ,if you speak up and make it known to the CO,DO,or HQ they will be forced to watch their selves from here on out .